Learning by CHIR without Storing Internal Representations

A new learning algorithm for feedforward networks, learning by choice of intern al represent at ions (C HIR), was recently introduced [1,2]. W hereas many algor it hm s red uce th e learning proce ss to minimizing a cost function over t he weights, our method treats th e internal representations as the funda ment al ent it ies to be determi ned. T he algo rithm applied a sea rch procedure in the space of intern al representations, together wit h coop er ati ve adaptation of the weights (e.g ., by using perceptron learning). Tentative guesses of the in ternal representations, however , had to be sto red in memory. Here we present a new version , CHIR2, which eliminates the need to store internal repr esent at ions and at the same time is fast er than the original algo rithm . We first describ e a basic version of CI-IIR2, tailored for networks wit h a single output and one hidden layer. We tested it on three problems contiguity, symmet ry, and parity and compared its performan ce wit h baekpr opagation. For all these pr oblems our algorith m is 30100 times faster th an backpr opagation , and , most significan tly, learning time increases mor e slowly with system size. Next, we show how to modify the algori thm for networks with many outpu t uni ts an d more t han one hid den layer. T his version is tested on the combine d pa ri ty-l-symm etry problem and on the rand om asso ciations task . A third modification of the new algori thm , suit able for networks with binary weights (all weight s and thresholds are equal to ±1) , is also described, and t ests of it s performance on the parity and the ran dom teacher problems are repor ted .